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ABSTRACT 

To ensure the quality of digital simulation game, we utilized the usability evaluation heuristic in the design and 
development processes of e-cafe operational management game-based learning material for students. The application of 
usability evaluations during this study is described. Additionally, participant selection, data collection and analysis; and 
results relevant to usability portions of this study are presented. 
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1. INTRODUCTION 

Nowadays college and university students have never experienced a world without computers. Since digital 
simulation game is designed more visual, interactive, and focused on problem-solving (Pasin, & Giroux, 
2011). It is more attractive for students to learn the complex knowledge via digital simulation game for 
college students. Therefore, this study employed design-based research to examine the design and 
development of an educational game simulating the daily operational management of an “e-cafe”. This study 
employed the principle of “usability” to determine whether this material corresponds to the above principle 
and whether it can meet the expectations of learners as an education resource. 


2. LITERATURE REVIEW 


2.1 Digital Simulation Game 

Digital simulation games are used for education and training in a wide range of fields including the military, 
medical services, and scientific experiments. Unlike digital games, simulation games are usually less 
associated with fantasy. Simulation games involve players assuming roles through which they seek to resolve 
issues or accomplish missions (Charsky, 2010). Digital simulation facilitates learning within created 
scenarios while enjoying gameplay that greatly resembles situations in the real world. Following a series of 
game activities, learners respond to various scenarios while achieving learning objectives. Researchers have 
indicated that digital games with educational functions can exert a positive effect on learners with regard to 
motivation and performance (Tompson & Dass, 2000, Bai, Pan, Hirumi, & Kebritchi, 2012). 

2.2 Usability Testing 

Usability can reveal whether a given product serves the needs of users and whether the product can be 
operated intuitively. Nielson (1993) proposed five indicators for the evaluation of usability; learnability, 
efficiency, memorability, and error and satisfaction. These indicators can be used to determine whether users 
are able to browse a webpage without facing obstacles, whether the page can be accessed intuitively, and 
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whether the webpage provides information sufficient to satisfy the needs of users. The International 
Organization for Standardization (ISO) pointed out in 1998 (ISO 9241-11) that “Usability’ refers to the 
extent to which a product can be used by specified users to achieve specified goals with effectiveness, 
efficiency and satisfaction in a specified context of use.” The most common methods employed in the 
evaluation of usability include heuristic evaluation and usability testing. This study recruited subjects with 
the aim of revealing user problems, evaluating the usability of the educational simulation game, and 
providing the basis from which to revise teaching materials in the future. 


3. METHODOLOGY 

Our evaluation of usability involved both quantitative and qualitative methods, the collection of data through 
questionnaires, observation and interviews, and the compilation of opinions and suggestions for the design 
and usability of the e-cafe simulator. This study will expound the subjects, testing processes, and research 
tools as follows. 

3.1 Subjects and Testing Processes 

Ten participants were invited to take part in the testing of a simulation game with regard to usability. All 
participants were college students at a male/ female ratio of 3:2 and all were participating in this type of 
exercise for the first-time. Through feedback from the subjects, we came to realize the needs of users and 
revised the simulation game accordingly. Tests and interviews were conducted with regard to the content and 
usability of the game. 

3.2 Testing Tools 

Each participant was observed one-on-one in the usability testing process and Morae software was applied to 
videotape the entire process. Prior to testing, the objectives of the evaluation were outlined for participants, 
who were then requested to sign a consent letter. Throughout the testing process, the participants were closely 
observed and provided objectives (tasks) in accordance with their progress. Subjects were expected to 
accomplish each phase of their tasks in accordance with the statements provided by observers. Participants 
were expected to undertake eleven missions and observers kept a close eye to verify whether they were able 
to complete them on their own. 

3.2.1 Questionnaire on Task Difficulty 

The eleven tasks in this study were designed to determine whether the subjects were able to complete the 
tasks they were assigned in the digital simulation game. Questions were ranked on a scale of 1 to 5 with 1 
representing “very easy” and 5 representing “very difficult”. 

3.2.2 Questionnaire on User Experience 

This study adopted the TAM model to measure the responses of participants following completion of the 
simulation game. This questionnaire comprised a total of 22 questions in four dimensions: attitude, perceived 
ease of use, intention to use, and perceived usefulness. Questions were ranked on a scale of 1 to 5 with 1 
representing “strongly disagree” and 5 representing “strongly agree”. 

3.2.3 Interview Outline 

Following completion of the usability test, interviews were conducted to determine the feelings of 
participants. The four interview questions were as follows: 

Ql: Did you encounter any problems when undergoing tasks in the game? How did you resolve them? 

Q2: Did you better comprehend the process of running a cafe after completing the “e-cafe” simulation game? 
Q3: Please outline the concepts related to business administration that you learned from the game? 

Q4: Do you think the “e-cafe” management simulation game helped you to acquire knowledge about 
business administration? 
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4. RESULTS 


4.1 Result Analysis for the Questionnaire on Tasks Difficulty 

The questionnaire was meant to determine how the subjects felt about each mission. For the eleven question 
items, the average score was 1.8 (SD=0.93). The scale of difficulty ranged from 1 (very easy) to 5 (very 
difficult). The participants felt that task 8 was the most difficult to accomplish (M=2.7), followed by task 10 
(M=2.2), task 6 (M=2.1) and task 7 (M=2.1). Task 8 involved adjusting the shift schedule. Task 10 required 
that participants organize staff training sessions. Task 6 required subjects to improve the quality of raw 
materials. Task 7 required subjects to devise a marketing approach to expand the customer base. In task 5, 
participants were required to adjust the selling price. 

4.2 Result Analysis for Game User Experience 

Questions were ranked on a scale of 1 to 5, with 1 representing “strongly disagree” and 5 representing 
“strongly agree”. As shown in Table 4, the average score was 3.5 (SD=0.86), suggesting that participants had 
positive attitudes toward the simulation game. In the “Attitude” sub-category, the four questions averaged 3.7 
(SD=1.01), implying that most of the subjects enjoyed the simulation game. In “Perceived ease of use,” the 
five questions averaged 3.44 (SD=0.88), which indicated that participants felt the game was easy. In 
“Intention to use,” the three questions averaged 3.9 ( SO — 0.85), and each sub-question reached 3.8 or more, 
indicating that subjects strongly accepted the simulation game. The average score for sub-question 3 was 4.1 
(SD=0.74), indicating that subjects were keen to participate in simulation games. In “Perceived usefulness,” 
the five questions averaged 3.36 (SD=0.8), indicating that participants felt the digital simulation game is 
useful for them. 

4.3 Result Analyses of Interviews 

Interview analysis was meant to reveal the experience of users and examine areas that were not adequately 
covered by questionnaires or observation. Interview analysis related to the questions is as follows: 

Ql. Did you encounter any problems when undergoing missions in the game? How did you solve it? 
Ten participants outlined problems they encountered when playing the simulation game. Three subjects 
did not know that the staffer’s name could be clicked. Two of the participants were unable to differentiate the 
faces of staffer and felt that a failure to mention the names of staffers made it difficult to arrange shift 
schedules. One participant did not know that the page could be changed from the counter to the guest area 
simply by clicking and kept wondering why the coffee machine could not be clicked to make repairs. 
However, most of these were resolved after the subjects tried several times. Clearly, the simulation game 
requires the means to explain all of the game functions. Nonetheless, these observations elucidated the needs 
and expectations of the subjects, such that the game could be improved in the future. 

Q2. Did you better comprehend the process of running a cafe after completing the “e-cafe” 
simulation game? 

Nine out of ten subjects (except for S8) believed that the simulation game helped them to understand how 
to manage a cafe. 

Q3. Please outline the concepts related to business administration that you learned from the game? 

All subjects felt that they developed their skills related to business administration from the simulation 
game. These skills included cost control, arranging shift schedules, picking the location in which to open a 
shop, the management of raw materials, equipment repair and maintenance, profit, and procurement. 
Nonetheless, a number of other opinions were also expressed: 

SI: It appears to me that hands-on experience would be more helpful. S2: I never took any class in 
business administration before, so I cannot say for sure. As to concepts, I think the most helpful ideas 
were about management, shift schedules, and the timing of equipment maintenance/repair. S4: I believe 
that cost control and customer satisfaction were dealt with effectively in the game. 
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Q4. Do you think that the e-cafe management game was of any help to you in acquiring 
knowledge in business administration? 

Eight subjects believed that the simulation game could be used to complement business administration 
curriculum. However, SI and S8 disagreed. 

S5: Yes. (The researcher asked about why.) Ugh, seems that we can have a better time in the classroom. 
S6: Yes, I enjoyed playing the game. S6 felt that if this game was played during class, they would not 
need to peruse reading materials. S8: I have never taken any classes in business administration; 
however, I am left wondering if this would be too simple to be fit into the real world. For example, staff 
affairs appeared too simplified. Raw materials, prices, and the quality of materials appeared more 
authentic. 


5. DISCUSSION AND CONCLUSION 

We conducted an examination of the usability of the “e-cafe management digital simulation game” as well as 
the test results and feedbacks from the participants. Our aim was to identify the areas corresponding to the 
expectations, requirements, and suggestions of users for use as a reference in the development of educational 
simulation games. 

With regard to usability, every participant presented a positive attitude toward every dimension of the 
e-cafe simulation game. The mission observation and the scale of difficulty perceived by participants were 
matched, as subjects encountered difficulties in the complex missions but felt that most of the missions were 
quite easy. Interviews revealed that subjects regarded the game as a valuable tool in developing the skills 
required for shop management. We believe that the reason for the strong acceptance of the simulation game 
was its overall design, which accurately represented a real coffee shop. It is our hope that this study provides 
a valuable resource for the development of digital simulation games as an educational resource. 


ACKNOWLEDGEMENT 

This study was supported by the National Science Council, Taiwan, under grant numbers NSC 100-25 1-S- 
032-003-MY3. 

REFERENCES 

Bai, H., Pan, W., Hirumi, A., & Kebritchi, M. (2012). Assessing the effectiveness of a 3-D instructional game on 
improving mathematics achievement and motivation of middle school students. British Journal of Educational 
Technology, (43) 6, 993-1003. doi: 10. Ill 1/j. 1467-8535.201 1.01269.x 

Charsky, D. ( 2010) From Edutainment to Serious Games: A Change in the Use of 

Game Characteristics. Games & Culture. 5(2), 177-198. 

Davis, F.D., (1989). ‘Perceived usefulness, perceived ease of use, and user acceptance of information technology’. MIS 
Quarterly,13(3) ,319-340. 

Davis, F.D., Bagozzi, R.P., & Warshaw, P.R.(1989) User Acceptance of Computer Technology:A Comparison of Two 
Theoretical Models. Management Science, 35(8), 982-1003. 

Huang, W. H. (2010). Evaluating learners' motivational and cognitive processing in an online game-based learning 
environment. Computers in Human Behavior, doi: 10. 1016/j.chb. 2010.07. 021 

Pasin, F. & Giroux, H. (201 1). The impact of a simulation game on operations 

management education. Computers & Education, 57, 1240-1254. 

Tompson, G. H., & Dass, P. (2000). Improving students’ self-efficacy in strategic management: the relative impact of 
cases and simulations. Simulation and Gaming, 31, 22 — 4 1 . 


364 



